Papers with language modeling training
Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) excel on English NLU tasks, yet struggle to extend their NLU capabilities to underrepresented languages. |
| Approach: | They integrate machine translation models (MT) directly into LLM backbones via sample-efficient self-distillation. |
| Outcome: | The proposed model outperforms translation-test models on 127 low-resource languages. |
Sustainable Modular Debiasing of Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing debiasing methods modify all of the PLM parameters, which is costly and leads to (catastrophic) forgetting of useful language knowledge. |
| Approach: | They propose a modular debiasing approach based on dedicated adapters that inject adapter modules into the original PLM layers and update only the adapters. |
| Outcome: | The proposed approach is based on dedicated adapters and retains fairness even after large-scale training. |